compIAM package#

This tutorial built around the compIAM package (computational analysis of Indian Art Music, and programatically stylized as compiam), which is a collaborative initiative involving many researchers that aims to put together a common repository of datasets, tools, and models for the computational analysis of Carnatic and Hindustani music.

Motivation#

Tons of very interesting and useful works have been carried out within the topic of computational analysis of Carnatic and Hindustani Music. However, it is common to come across, in such research contexts, difficulties and problems in finding or reproducing the published research.

compiam aims to be a common and community-driven repository including Python-based and standardized implementations of research works for the computational analysis of IAM. In addition, compiam includes tools to easily access relevant datasets and corpora.

Note

compiam is very open to contributions! To integrate your works or relevant implementations from other researchers please check out the contributing guidelines.

Installation#

compiam is registered to PyPI, therefore the latest release can be installed with:

pip install compiam

Nonetheless, to get the latest version of the library with the fresher updates, proceed as follows:

git clone https://github.com/MTG/compIAM.git
cd compIAM

virtualenv -p python3 compiam_env
source compiam_env/bin/activate

pip install -e .
pip install -r requirements.txt

Available components#

compiam includes the following components:

What to load?

Description

Datasets

Initializing dataset loaders through mirdata (see Datasets)

Corpora (Dunya)

Accessing the Dunya corpora (see Corpora)

Tools and models

Initializing tools and non-pre-trained ML and DL models

Pre-trained DL models

Initializing pre-trained models

Basic usage#

compiam does not have terminal functionalities but it is to be used within Python based-projects. First, import the library to your Python project with:

## Importing compiam to the project
import compiam

We will import some additional util packages for improved user experience along the tutorial.

import os

import numpy as np
import IPython.display as ipd

from pprint import pprint

import warnings
warnings.filterwarnings('ignore')

Importing the integrated tools#

The integrated tools and models are organized by:

  • First, the following fundamental musical aspects: melody, rhythm, structure, and timbre.

  • Within each of the musical aspect, the tools then are grouped by task.

Print out the available tasks for each category.

pprint(compiam.melody.list_tasks())
['pattern', 'pitch_extraction', 'raga_recognition', 'tonic_identification']

You may also print out the available tools for a musical aspect.

pprint(compiam.melody.list_tools())
['pattern.CAEWrapper*',
 'extraction.segmentExtractor',
 'pitch_extraction.FTANetCarnatic*',
 'pitch_extraction.Melodia',
 'raga_recognition.DEEPSRGM*',
 'tonic_identification.TonicIndianMultiPitch']

Finally, you may be only interested on listing only the tools for a particular task.

pprint(compiam.rhythm.transcription.list_tools())
['MnemonicTranscription']

Right, we have now learned how to list the available tools in compiam. Initializing and using the integrated works in compiam can be done by basically importing these from the respective module. Use that to load non-trainable tools and non-trained machine or deep learning (ML/DL) models.

from compiam.melody.tonic_identification import TonicIndianMultiPitch
from compiam.rhythm.transcription import FourWayTabla

Each tool includes specific methods and attributes, so make sure to carefully check the compiam documentation before using a particular tool. However, integrated tools all include a .extract() (for heuristic-based tools) or a .predict() (for data-driven tools) to directly run the tool on any input audio.

Wrappers#

compiam also includes wrappers to easily initialize relevant datasets, corpora, and also pre-trained models for particular problems.

Wrapper

Description

Option list

compiam.load_dataset()

Initializing dataset loaders

Run compiam.list_datasets()

compiam.load_corpora()

Accessing the Dunya corpora

Run compiam.list_corpora()

compiam.load_model()

Initializing pre-trained models

Run compiam.list_models()

Why wrappers?#

Wrappers can be directly imported from compiam, and serve as a fast and standardized manner to access the data and pre-trained models in compiam. Let us showcase the nice utility of the compiam wrappers.

from compiam import load_dataset, load_corpora, load_model
The .load_dataset() wrapper#

We will start by listing the available datasets through compiam using the .load_datasets() method which is implemented in the core of compiam.

pprint(compiam.list_datasets())
['saraga_carnatic',
 'saraga_hindustani',
 'mridangam_stroke',
 'four_way_tabla',
 'compmusic_carnatic_rhythm',
 'compmusic_hindustani_rhythm',
 'compmusic_raga',
 'compmusic_indian_tonic']

The compiam.load_dataset() wrapper allows the user to initialize the dataloader for any dataset included in the list displayed above. Let’s take a look at an example.

## Initializing dataloader for an example dataset
fw_tabla_dataset = load_dataset("four_way_tabla")
fw_tabla_dataset
The four_way_tabla dataset
----------------------------------------------------------------------------------------------------


Call the .cite method for bibtex citations.
----------------------------------------------------------------------------------------------------


Four-Way Tabla track class

    Args:
        track_id (str): track id of the track
        data_home (str): Local path where the dataset is stored.

    Attributes:
        track_id (str): track id
        audio_path (str): audio path
        onsets_b_path (str): path to B onsets
        onsets_d_path (str): path to D onsets
        onsets_rb_path (str): path to RB onsets
        onsets_rt_path (str): path to RT onsets

    ----------------------------------------------------------------------------------------------------

Dataset loaders provide a vast amount of advantages and introducing these into our research pipelines may be very beneficial. compiam relies on mirdata, a Python library focused on common dataloader for MIR datasets. So bear in mind that when loading a particular dataset using compiam.load_dataset(), we are basically initializing a mirdata dataloader. To get a more insightful walkthrough of the dataset loaders and how these work, please see Datasets, and make sure to check out and star mirdata.

The .load_corpora() wrapper#

compiam provides access to both Carnatic and Hindustani [SKG+14] corpora from Dunya. Initialize the access tool to these huge collections of data using the designated wrapper.

## Initializing dataloader for an example dataset
carnatic_corpora = load_corpora("carnatic", cc=True, token="your-token-goes-here")

Warning

A complete walkthrough of the corpora access in this book is not possible because of the explicit need of a unique, personal, and intransferible access token. Please get your token and check the detailed documentation we have put together. Feel free to reach out in case you encouter issues or questions.

The .load_model() wrapper#

For the case of ML/DL models that are data-driven and therefore trained using particular collections of data, we may use the compiam.load_model() wrapper to load the tool as you would do using from compiam.<musical-aspect>.<task> import <particular-model>. However, loading the model using the wrapper will load by default the pre-trained model given publicly available weights and checkpoints for the integrated implementation.

Tip

When listing available tools using the list_tools() functions, some will appear with a * at the end. That is meant to indicate that such tools have pre-trained models available, which may be loaded using the wrapper compiam.load_model().

You may obviously initialize the untrained version of the model and load the weights afterwards, especially if you have your own checkpoints or seek to train the model yourself. However, the compiam.load_model() wrappers offers a handy and rapid way to load a pre-trained model for a particular task. The weights for the integrated models are stored (and included in the release of compiam) in compiam/models/<musical-aspect>/<model-name>/ (writing the path UNIX-styled here!).

Important

Only the ML and DL-based models with available weights can be loaded using the .load_model() wrapper. The non-trainable tools or the models with no available weights must be imported from their corresponding module.

Let’s list the available pre-trained ML/DL models through compiam.

pprint(compiam.list_models())
['melody:deepsrgm',
 'melody:ftanet-carnatic',
 'melody:cae-carnatic',
 'structure:dhrupad-bandish-segmentation']

This list includes all the models in compiam that can be loaded using a set of publicly available weights. Let’s now run a snippet to show how fast we can expriment through the use of the wrappers in compiam.

%pip install tensorflow==2.9.3

Note

Some tools require specific dependencies that are not installed by default. These dependencies are specified in the online compiam documentation. In addition, if a tool is loaded and a dependency is missing, an error message will be thrown requesting the user to install the missing package.

# Loading pre-trained model for prediction
ftanet_carnatic = load_model("melody:ftanet-carnatic")

saraga_carnatic = load_dataset(
    "saraga_carnatic",
    data_home=os.path.join("..", "audio", "mir_datasets"),
)
2022-12-03 21:06:55.554824: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/cv2/../../lib64:
2022-12-03 21:06:55.554861: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-12-03 21:06:57.656140: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/cv2/../../lib64:
2022-12-03 21:06:57.656285: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-12-03 21:06:57.656329: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fv-az436-912): /proc/driver/nvidia/version does not exist
2022-12-03 21:06:57.656581: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

These methods in the following cell download the dataser for you, and make sure no files are missing or have been corrupted during the downloading/unzipping process.

#saraga_carnatic.download()
#saraga_carnatic.validate()

Note

Saraga Carnatic is about 24GB of data and therefore we will not download it within the context of this tutorial.

An audio example from the dataset has been selected for tutoring purposes: “Sri Raghuvara Sugunaalaya” performed by Sanjay Subrahmanyan. The id of this track in the Saraga Carnatic dataloader is "109_Sri_Raghuvara_Sugunaalaya".

# Getting audio from random track in the dataset
saraga_tracks = saraga_carnatic.load_tracks()
our_track = saraga_tracks["109_Sri_Raghuvara_Sugunaalaya"]

# Predict!
predicted_pitch = ftanet_carnatic.predict(our_track.audio_path)
2022-12-03 21:07:18.430140: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 83886080 exceeds 10% of free system memory.
2022-12-03 21:07:18.498995: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 83886080 exceeds 10% of free system memory.
2022-12-03 21:07:18.600875: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 83886080 exceeds 10% of free system memory.
2022-12-03 21:07:18.643197: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 83886080 exceeds 10% of free system memory.
2022-12-03 21:07:18.686142: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 83886080 exceeds 10% of free system memory.
1/1 [==============================] - ETA: 0s

1/1 [==============================] - 11s 11s/step
1/1 [==============================] - ETA: 0s

1/1 [==============================] - 8s 8s/step
1/1 [==============================] - ETA: 0s

1/1 [==============================] - 10s 10s/step

Util functions#

compiam also includes visualisation and general util functions to assist the comprehensive computational study of musical repertoires and issue quite a number of operations that are useful for the said research field. Check the visualisation documentation and the utils documentation out for a detailed presentation of the available tools.

We will now use the .plot_pitch() method which allows us to plot the pitch curve we have just extracted from the audio example.

from compiam.visualisation.pitch import plot_pitch

# Plotting waveform
plot_pitch(
    pitch=predicted_pitch[:, 0],
    times=predicted_pitch[:, 1],
);

# Playing 10 seconds of audio used for prediction
audio, sr = our_track.audio
audio_short = audio[:10*sr]
ipd.Audio(
    data=audio_short,
    rate=sr
) 
../_images/compiam_28_1.png

The extent of compiam#

In the very first launch of compiam (v0.1.0), the package includes access to 8 datasets, access to the Carnatic and the Hindustani corpora in Dunya, and a bit more that 10 tools and models to use, covering 8 different tasks. We are aware, and we are very interested in bringing that into the open, that the topic of the computational analysis of Indian Art Music, especially on the tools and models side, is way broader. The literature includes a vast amount of very interesting and relevant works that, mainly because of the time constraint, are not included in the first release of compiam.

However, we have been designing compiam so that it evolves as a community-driven initiative, in which anyone can integrate datasts, tools, and models for the computational analysis of Indian Art Music.

Therefore, this tutorial is also a great chance to present the official launch of compiam, and engage researchers and developers interested in the research topic of computational analysis of Carnatic and Hindustani music to contribute to this library. The ultimate goal of this software is that the relevant collections of data and versatile tools for this topic are made available an easy accessible for everyone, while boosting the visibility and community usage of these.

Contributing#

compiam is continuously evolving and it is very much open to contributions! We have put together complete set of contribution guidelines to help you improving and/or expanding the library. Potential contributions are:

  • Adding new implementations of tools or models.

  • Adding new datasets by first adding a dataloader to mirdata (see Datasets) and then link it to compiam (we do have mirdata maintainers in the team therefore close help should be given).

  • Fix, improve, update, or expand existing implementations.

  • Fixing bugs.

  • Improving documentation.

Don’t hestitate to reach out or open an issue in the compaim GitHub repository to request for further help or raise questions.